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ABSTRACT 

Educational games have the potential to be innovative forms of 
learning assessment, by allowing us to not just study their 
knowledge but the process that takes students to that knowledge. 
This paper examines the mediating role of players’ moves in 
digital games on changes in their pre-post classroom measures of 
implicit science learning. We applied automated detectors of 
strategic moves, built and validated from game log data combined 
with coded videos of gameplay of 69 students, to a new and larger 
sample of gameplay data. These data were collected as part of 
national implementation study of the physical science game. 
Impulse. This study compared 213 students in 21 classrooms that 
only played the game and 180 students in 18 classrooms in where 
the players’ teacher used game examples to bridge the implicit 
science learning in the game with explicit science content covered 
in class. We analyzed how learning outcomes between conditions 
were associated with six strategic moves students made during 
gameplay. Three of the strategic moves observed are consistent 
with an implicit understanding of Newton’s First Law, the other 
three strategic moves were not. Path analyses suggest the 
mediating role of strategic moves on students’ implicit science 
learning is different between the two conditions. 
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1. INTRODUCTION 

Digital games are gamering increasing attention as potential 
learning environments as the volume of research increases 
indicating games may foster scientific inquiry, problem-solving, 
and public participation in breakthrough scientific discoveries [1]. 
Because nearly all youth and many adults participate in Internet- 
based games [2], educators and researchers are trying to tap this 
pervasive vehicle for learning and assessment environments for 
the 21st century [3]. 

Our research group studies how games can be used to improve 
learning of fundamental high-school science concepts (e.g. 
Newton’s laws of motion). Our games use popular game 
mechanics embedded in accurate scientific simulations so that 
through engaging gameplay, players are interacting with digitized 
versions of the laws of nature and the principles of science. We 
hypothesize that as players dwell in scientific phenomena, 
repeatedly grappling with increasingly complex instantiations of 
the physical laws, they build and solidify their implicit knowledge 
over time. 

It is not our intent that these games teach science content 
explicitly, but rather that they engage the learner with scientific 
phenomena allow them to build their implicit understandings 
about these phenomena through gameplay. To measure implicit 
learning in games, we built automated detectors of strategies we 


saw players using in the games [4, 5]. Thus, we address the 
question: Do learners ’ strategic moves in the game correspond to 
increased implicit understanding of the science content outside 
the game? 

We also examine the role of the teacher in game-based learning. 
As Jim Gee points out, games rely on what he refers to as the Big 
“G” Game - the surrounding interactions that arise because of and 
support the game [6]. Post-game debriefing and discussions 
connecting gameplay with classroom learning are critical in 
helping students apply and transfer learning that takes place in 
games [7]. Our research attempts to capture the strategies players 
develop during gameplay that may reveal implicit knowledge, so 
that we can help educators seize and leverage that implicit 
learning to support explicit classroom learning. 

Success in this approach will result in a new way to think about 
game-based assessments, starting not from prescribed learning 
outcomes, but from watching what types of strategy development 
actually take place. The final step of this research, reported in this 
paper, is to examine the extent to which strategic moves used 
while playing Impulse mediate changes in classroom measures of 
students’ understanding of the same science content. 

2. GAME. IMPULSE 

The game Impulse is built for the web and wireless devices. 
Impulse challenges players use an impulse (a click or touch on the 
screen) to move their ball to a goal without crashing into any other 
(ambient) balls on the screen. All the balls have mass and obey 
Newton’s laws of motion. As the levels of the game increase, 
more ambient balls are introduced, with varying mass. 

Impulse is an attempt by designers to immerse a player in what is 
known to physicists as a n-body simulator. We hypothesize that 
by having to predict the motions of the particles, and their 
reactions to the force imparted by the impulse, the player will 
build implicit knowledge of forces and motions (Figure 1) that we 
could measure through data mining. 

The first 20 levels of the game introduce players to 4 particles of 
different mass, providing 5 levels of experience with each of the 4 
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particles; across these 5 levels, the number of particles in the 
game space increases from 1 to eventually 10. Beginning in Level 
21, players encounter particles with different masses 
simultaneously. As players reach higher levels with greater 
numbers and variety of masses of particles, they need to “study” 
the particles’ behavior to predict the motion of particles so that 
they can guide their particle to the goal, not run out of energy, and 
avoid collision with other particles. 

3. STRATEGIC MOVES 

Our research attempts to capture and automatically assess the 
range of strategies players develop during gameplay. We 
identified a set of 6 strategic moves that we observe players 
making in the game Impulse (Table 1). Three of these strategic 
moves are theorized to constitute evidence of implicit 
understandings of Newton’s First Law: each particle will keep 
moving on its path without an impulse or force from another 
particle. The remaining three strategic moves reflect an 
understanding of the game mechanic, but are not considered 
strong evidence of implicit understanding of Newton’s First Law. 


Table 1. Strategic moves and coding deHnitions 


Strategic 

Move 

Coding Definition 

* Float 

The player particle was not acted upon for more 
than 1 second 

Toward goal 

The learner intended to move the player particle 
toward the goal 

* Stop/slow 
down 

The learner intended to stop or slow the motion 
of the player particle 

*Player path 
clear 

The learner intended to move non-player 
particles to keep the path of the player particle 
clear 

Goal clear 

The learner intended to move non-player 
particles to keep the goal clear 

Buffer 

The learner intended to create a buffer between 
the player and other particles to avoid collision 


*Evidence of implicit understanding of Newton’s First Law 


Video data was collected from 69 high school students, to develop 
automated detectors of these strategies. Every click in randomly 
selected, three-minute video segments, one per student, was coded 
for these strategic moves, with every player action in these video 
segments coded as to which strategy it represented. Two coders 
coded ten videos with Kappa values exceeding 0.70 for all of 
these strategic moves [4, 5]. 

We built classifiers to infer the ground truth labels created by the 
video coders. For each player action a set of 66 features of that 
action were automatically distilled, including the time since the 
last player action and the distance between the player particle and 
goal. These features were then aggregated at the click level to map 
to the labels provided by the video coders [6]. Classifiers were 
created using J48 decision trees within RapidMiner 5.3 that 
mapped the student behaviors in the features distilled from the 
clickstream data to the training labels, cross-validating at the 
student level. All detectors discussed here had cross-validated 
Kappas between 0.51 and 0.86 and A’ between 0.78 and 0.97 [6]. 

4. IMPLEMENTATION STUDY 

Having developed these detectors of student strategic moves, we 
then collected a much larger data set to be able to study the 
relationship between in-game strategic moves, pedagogical 
practices, and learning outcomes. To this end, we conducted an 
implementation study [8] to examine the conjecture that implicit 


learning in game play can help prepare students for classroom 
learning. 

Forty- two teachers were assigned to one of three groups (14 per 
group). Teachers could include a maximum of three sections of an 
individual class. Of the 42 teachers who initially agreed to 
participate, 23 teachers completed the study (55 percent), resulting 
in this final sample with complete data: 

Bridge: 180 students in 18 classes in which 8 teachers 
incorporated game examples to bridge game play and science 
content 


Game Only: 213 students in 21 classes in which 10 teachers 
encouraged students to play the game, but provided no in-class 
interaction around the game 


Control: 108 students in 11 classes in which 5 teachers taught the 
science content as they normally do, without games. 

Students took pre-post online assessments with six items, three 
dealing with Newton’s First Eaw and three dealing with Newton’s 
Second Law. All items were written to be answerable with an 
intuitive understanding of the physics concepts and were piloted 
with think-aloud interviews. Both assessments had a maximum of 
10 points possible. Assessment scores were standardized as Z- 
scores and all coefficients are reported in effect sizes. 


Hierarchical linear modeling of data from the 23 teachers (50 
classes) shows a significant positive effect of the Bridge and 
Game Only groups compared to the Control group on student’s 
post-assessment scores after accounting for pre-assessment scores 
[8]. This group effect, however, was significantly moderated by 
whether or not the class was a Honors/ AP class (Figure 2). There 
was also a significant main effect for gender, with female students 
receiving lower post-scores than male students. 



Figure 2: Predicted post-assessment scores across study 

conditions in Honors/AP classes versus non-Honors/AP classes 
(y-axis=standard deviations from the mean post-score, 
accounting for all components of the HLM model) [8]. 

The group effect was significant among students in non- 
Honors/AP classes. Among students in Honors/AP classes. Bridge 
students performed better than Game only students but not 
Control students. These results, while intriguing, tell us that the 
Bridge condition was generally best, but do not explain why 
Bridge was better. Did the teachers in the Bridge condition 
promote learning separate from the game? Or did it actually drive 
different behavior within Impulse, making the game a more 
valuable learning experience? 
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5. THE ROLE PLAYED BY IN-GAME 
STRATEGIC BEHAVIOR 

The final step in this researeh, and the speeifie eontribution novel 
to this paper, is to eonneet in-game measures of implieit seienee 
learning with external measures of those eoneepts. Speeifieally, 
we hypothesize that strategie moves eonsistent with an implieit 
understanding of Newton’s First Law will mediate ehanges in 
these external assessments, whereas the other strategie moves will 
not be assoeiated with ehanges in the pre-post assessments. 

5.1 Apply Automated Detectors 

We applied the automated deteetors built with the sample of 69 
students to this larger sample of gameplay data from 393 students 
to deteet when learners used eaeh type of strategie move. The 
deteetors were applied to every student aetion during the entire 
duration of gameplay, 1.01 million aetions in total. The same log 
data features were automatieally distilled for this entire data set as 
for the initial ereation of the models. Then this data was inputted 
into RapidMiner 5.3, along with the previously generated W-J48 
deeision trees model files, in order to apply the trees to the data. 
The result was a predietion for every eliek, for eaeh of the 
relevant strategie moves in Table 1, of the deteetor’s eonfidenee 
that strategy was being used. Every learner aetion in this game 
was thereby annotated with an estimated probability that the 
learner was using eaeh of the strategie moves. 



Figure 3: Average probability for each strategic move (y-axis) 
by game level (x-axis) 

Figure 3 shows the average probability for eaeh strategie move at 
eaeh game level. The most prevalent strategie moves were 
Toward Goal and Float, with Float being evidenee for implieit 
understanding of Newton’s First Law. The least eommon strategie 
moves were Stop/Slow Down (evidenee for implieit 
understanding) and Buffer. Float refleets the absenee of aetivity 
(on the player partiele in the time prior to the eliek and ean eo- 
oeeur with any other strategie move. Stop/Slow Down, in 
eontrast, refleets a deliberate attempt by the player to stop or slow 
down the motion of the player partiele. Float and Stop/Slow 


Down both refleet understandings of Newton’s First Law e.g., a 
mass will keep moving until aeted upon by a foree, but the float 
strategy is a passive move and the stop strategy is an aetive move. 

Figure 3 also shows evidenee of shifts in behavior every 5 levels. 
The eyelieal patterns in this data eorrespond with the planned 
transitions in the game. Every 5 levels, the game reduees the 
diffieulty level of the game when a new ehallenge (e.g., partiele 
with a different mass, two partieles with different masses) is 
introdueed, by deereasing the number of partieles in the spaee (a 
deerease in gameplay ehallenge whieh balanees for the inerease in 
eoneeptual ehallenge). However, the reduetion in the number of 
partieles makes it more likely a player will simply push the 
partiele toward the goal, leading to eorresponding deelines in all 
of the other strategies. Overall, as the number of partieles in the 
game spaee inereases, the average probability of using the simple 
Toward Goal strategy deelines while the probabilities of using the 
other strategies inerease. 

5.2 Path Models 

Path models were built to estimate the mediating role of eaeh 
strategie move between prior aehievement and post assessment 
seores using SmartPLS [9]. As pre-assessment seores and 
Honor s/AP enrollment were signifieantly eorrelated, they were 
eombined into a single latent variable labeled Trior 
Aehievement’. Separate path models were ereated for the Bridge 
and Game Only eonditions (Figures 4 and 5). The standardized 
eoeffieients appear on the paths and the adjusted values appear 
in the eireles. T-values were ealeulated using a bootstrapping 
proeess with 1000 samples. 

Among students in Bridge elassrooms, the use of the Buffer 
strategy signifieantly mediates the impaet of prior aehievement 
and gender on post-seores (adjusted R^ = 0.151, p=0.005). This 
suggests using the Buffer strategy enhaneed Bridge student’s 
understanding of the eoneepts, beyond what is aeeounted for their 
prior levels of aehievement. In Game Only elassrooms, student 
use of the Buffer (adjusted R2 = 0.095, p=0.018). Stop (adjusted 
R^ = 0..149, p<0.001), and Float (adjusted R^ = 0.109, p=0.031), 
strategie moves signifieantly mediate the relationship between 
prior aehievement & gender on post-seores. In these elasses with 
no teaeher seaffolding of the gameplay, use of the Buffer and 
Float strategies enhaneed student’s understanding, but use of the 
Stop strategy diminished their understanding. 



Figure 4: Full path model — Bridge Classrooms 


Proceedings of the 8th International Conference on Educational Data Mining 


434 




Figure 5: Full path model — Game Only Classrooms 


In Bridge and Game Only elassrooms, onee gender differenees in 
strategie moves are taken into aeeount, the impaet of being a 
Female student on post-seores is no longer signifieant (eoeff=- 
0.127 in Bridge elassrooms, p=0.108; eoeff=-0.021 in Game Only 
elassrooms, p=0.759). This suggests the gender main effeet found 
in the HLM analyses may be entirely attributable to gender 
differenees in gameplay. One potential explanation is that while 
females play games at equal rates as males [2], the types of games 
they play and the amount of time they spend doing so may vary. 
Sueeess in a rapid-fire, reaetion time edueational game like 
Impulse may require gameplay skills more eongruent with games 
more popular among males (e.g, first person shooters) than the 
soeial, puzzle, and role-playing games females tend to prefer [2]. 

6. DISCUSSION 

It is noteworthy that two of the three strategies we antieipated 
refleeting an implieit understanding of Newton’s First Law were 
signifieant mediators in Game Only elassrooms. Player Path 
Clear, a strategie move applied to non-player partieles, may not 
have been a signifieant mediator beeause it is likely to eo-oeeur 
with Float, a strategie move applied to the player partiele. By 
eontrast, the other strategies were not signifieant mediators with 
one exeeption: Buffer, the simultaneous use of foree on more than 
one partiele when the partieles were in elose proximity to eaeh 
other. Sometimes those forees were in direet opposition to the 
other partieles (i.e., simultaneous use of the Stop strategy), while 
other times they were not. While Buffer was not a strategie move 
we a priori identified as eonsistent with an understanding of 
Newton’s First Law, these results suggest it plays a mediating role 
similar to Stop and Float. Use of the Buffer strategy was 
assoeiated with higher post-seores in Bridge and Game Only 
elassrooms. 

The negative mediating relationship of the Stop strategy in Game 
Only elassrooms is eonsistent with the HLM findings shown in 
Figure 2, where students in Honors-AP elasses did not perform on 
the post-assessment as well as students in non-Honors/AP elasses 
[8]. This laek of use of the Stop strategy is eonsistent with the laek 
of understanding of Newton’s Laws exhibited on the pre-post 
assessments. This suggests that learners who already have a basie 
understanding of the seientifie eoneepts may not be aided by the 
game as a sole intervention. Their improvement in seienee 
understanding is enhaneed when the game and the teaeher bridge 
materials are used together. These results reinforee the importanee 


of teaehers providing bridges between gameplay and seienee 
eontent. 

This paper also makes an important eontribution to the spaee of 
problems that ean be addressed by EDM. Many projeets have 
attempted to deteet strategie behavior in online learning. This 
projeet, by deteeting strategie behavior explieitly eonneeted to 
eore eoneepts, and modeling how different elassroom aetivities 
influenee in-game behavior, shows how EDM methods ean bridge 
understanding of the relationship between what students learn in 
elass, and how they behave online. As sueh, we are able to see the 
eonerete impaet of elassroom aetivity on gameplay behavior, and 
to measure its seope and manifestations. 

In the long term, then, this eombination of methods - automated 
deteetors, path analysis, and elassroom studies - ereates the 
potential to make EDM useful for investigating interventions not 
just online, but in elassroom settings as well. 
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